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1  Introduction 


This  document  provides  a  broadly  accessible  overview  of  the  Integrated  Cognitive-neuroscience 
Architectures  for  Understanding  Sensemaking  (ICArUS)  Phase  1  challenge  problem.  The  pages 
include  screen  shots  (denoted  by  black  rectangles)  from  the  ICArUS  Phase  1  Test  and  Evaluation 
(T&E)  Graphical  User  Interface  (GUI),  including  the  tutorial  that  participants  viewed  before  and 
during  experiments. 

This  document  is  not  meant  to  stand  alone.  For  a  detailed  description  of  the  ICArUS  Phase  1 
challenge  problem,  please  refer  to  the  following  document: 

Burns,  K.,  Greenwald,  H.,  &  Fine,  M.  (2014).  ICArUS  Phase  1  Challenge  Problem  Design 
and  Test  Specification.  MITRE  Technical  Report,  MTR140410. 

More  information  about  materials  developed  for  Phase  1  T&E  is  available  in  the  following 
document: 

Burns,  K.,  Fine,  M.,  Bonaceto,  C.,  &  Oertel,  C.  (2014).  Integrated  Cognitive-neuroscience 
Architectures  for  Understanding  Sensemaking  (ICArUS):  Overview  of  Test  and  Evaluation 
Materials.  MITRE  Technical  Report,  MTR140409. 

These  documents  are  available  at:  http://www.mitre.org/publications. 

For  reference  purposes,  Table  2  from  the  Phase  1  Challenge  Problem  Design  and  Test 
Specification  is  reproduced  below.  This  table  outlines  the  6  tasks  (also  known  as  "missions") 
that  comprise  the  primary  Phase  1  challenge  problem.  An  additional  Task  7,  outlined  in  Section 
9  below,  is  described  in  Appendix  A  of  the  Phase  1  Challenge  Problem  Design  and  Test 
Specification. 

In  general,  tasks  of  the  Phase  1  challenge  problem  require  predicting  which  of  several  enemy 
groups  is  most  likely  to  attack  a  given  location,  or  which  of  several  locations  a  group  will  most 
likely  attack.  Tasks  1-3  require  statistical  learning,  as  evidence  is  accumulated  over  time  from 
reports  of  individual  attacks.  Tasks  4-6  require  integrating  various  layers  of  intelligence,  as 
evidence  about  a  single  attack  is  provided  by  multiple  independent  sources  coincident  in  time. 
Task  7  combines  aspects  of  earlier  tasks  while  introducing  a  notion  of  self-initiated 
sensemaking. 

All  materials  used  in  human  experiments  were  approved  by  MITRE's  Institutional  Review  Board 
prior  to  use. 
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Task 

Description 

1 

Learning  two  groups.  [10  blocks  of  10  trials].  Attacks  by  two  groups  (A,  B)  are  generated  (SIGACTS),  one  on  each 
trial,  by  two  normal  distributions  -  each  with  a  different  base  rate,  center  location,  and  dispersion.  On  the  last  trial 
of  each  block  the  subject  is  shown  an  attack  location  and  asked  to  predict  the  probability  of  attack  by  each  group, 
{A%,  B%}.  The  subject  then  makes  a  forced  choice  (A  or  B),  to  allocate  resources  against  the  attack.  At  the  end  of 
the  trial  the  subject  sees  ground  truth  (A  or  B)  at  the  attack  location  and  he  reports  surprise  (on  a  7-point  Likert 
scale). 

2 

Learning  four  groups.  [5  blocks  of  20  trials].  Unlike  Task  1,  there  are  four  different  groups  (A,  B,  C,  and  D).  Also, 
before  reporting  {A%,  B%,  C%,  D%},  the  subject  is  required  to  draw  four  circles  -  each  representing  the  "two-to-one 
boundary"  for  a  group  (i.e.,  attack  by  the  group  is  twice  as  likely  inside  the  circle  compared  to  outside  the  circle). 

After  reporting  the  probabilities  {A%,  B%,  C%,  D%}  and  making  a  forced  choice  in  allocating  resources  against  one 
group,  the  subject  is  shown  ground  truth  (A,  B,  C,  or  D)  at  the  attack  location  and  he  reports  surprise. 

3 

Finding  centers.  [5  blocks  of  20  trials].  Unlike  Task  2,  the  task  is  to  find  group  centers  along  roads  "as  the  cow 
walks"  (rather  than  "as  the  crow  flies").  After  each  block  of  trials,  the  subject  clicks  four  points  A,  B,  C,  and  D  on  the 
roads  -  each  representing  the  "center  of  gravity"  for  a  group  (i.e.,  maximum  likelihood  location  for  a  group's 
center,  given  the  group's  attacks).  The  subject  then  reports  {A%,  B%,  C%,  D%}  at  the  attack  location.  After  reporting 
the  probabilities  and  making  a  forced  choice  in  allocating  resources,  the  subject  is  shown  ground  truth  (A,  B,  C,  or 

D)  at  the  attack  location  and  he  reports  surprise. 

4 

Scoping  attacks.  [10  trials,  2  stages  per  trial].  Unlike  Task  3,  the  subject  is  given  the  center  (from  HUMINT)  for  one 
group,  along  with  four  possible  locations  1,  2,  3,  and  4  of  attack  by  that  group.  In  stage  1,  the  subject  is  required  to 
estimate  the  probability  of  attack  at  each  location  based  on  a  normal  distance  decay  function  along  roads.  In  stage 

2,  the  subject  is  given  the  regional  boundaries  for  groups  (SOCINT),  along  with  the  inside/outside  region  attack 
likelihoods  from  PROBS.  The  subject  is  required  to  aggregate  the  prior  (HUMINT)  and  SOCINT  probabilities.  The 
subject  allocates  resources  in  percentages  (not  a  forced  choice)  across  the  sites,  and  is  shown  ground  truth  (1,  2,  3, 
or  4).  The  subject  reports  surprise  after  observing  ground  truth. 

5 

Fusing  layers.  [10  trials,  4  stages  per  trial].  Unlike  Task  4,  the  subject  is  given  an  attack  location  (SIGACT)  along  with 
group  centers  (HUMINT)  for  four  groups  (A,  B,  C,  D).  The  subject  is  also  given  the  probabilities  of  attack  by  each 
group  {A%,  B%,  C%,  D%},  based  on  the  HUMINT  distance  function.  The  subject  is  then  given  four  more  INTS  (IMINT, 
MOVINT,  SIGINT,  SOCINT),  one  at  a  time,  along  with  the  likelihoods  (from  PROBS)  for  features  of  INTS.  At  each 
stage  the  subject  updates  probabilities  {A%,  B%,  C%,  D%}.  After  the  last  stage  the  subject  allocates  resources  in 
percentages  {A%,  B%,  C%,  D%}  against  the  groups.  At  the  end  of  the  trial  the  subject  is  shown  ground  truth  and  he 
reports  surprise. 

6 

Choosing  layers.  [10  trials,  3  stages  per  trial].  Much  like  Task  5,  except  the  subject  decides  which  layer  of  INT 
(IMINT,  MOVINT,  SIGINT,  or  SOCINT)  to  receive  on  each  of  three  stages,  after  receiving  the  initial  HUMINT.  The 
criterion  for  each  choice  is  to  maximize  the  expected  amount  of  useful  information  to  be  gained  from  the  choice  of 
an  INT. 
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Purpose 

You  wi  II  bepiayrng  theroie  of  a  geospatial  intelligence  analyst  over  the 
course  of  seven  missions. 

In  each  mission,  you  must  make  judgments  concerning  the  attacks  by 
enemy  groups.  There  are  2-4  enemy  groups  that  are  act  tve  in  the  area  of 
interest  at  any  given  time.  They  are  known  as  Aqua,  Bromine*  Citrine*  and 
Diamond,  or  A,  B,  C,  and  D  for  short. 

Your  task  is  to  make  sense  of  va  rio  us  intei  I  igence  s  ources  i  n  order  to  predict 
which  group  is  most  likely  to  attack  or  which  location  a  group  v/iflattack. 
An  i  mportant  aspect  of  your  decision  making  w  ill  be  to  assess  the  relative 
chance,  or  probability,  of  attack  by  a ll  groups  in  the  area 

For  example*  fftherearefourgroupsthenyou  wiineedto  report  the 
probability  of  attack  by  each  group:  A,  B,  C*  D. 


Entering  probabilities 

You  w  i  II  enter  probabi  Ifcies  using  the  i  nt  efface  show  n  below.  There  are  3  methods  of 
entering  probabilities  you  may  choose  any  combination  of  methods  you  like. 

1.  You  can  use  the  mouse  to  drag  each  slider  up  or  down. 

2.  You  can  "cl  ick"  within  a  slider  to  automaticaty  move  it  to  that  point. 

3.  You  can  use  the  scrol  I  wheel  on  your  mouse  to  make  fine  a  djustments  w  ith  in  a  sider 

PI  ease  try  each  method  now. 


ip. 

□ 

■ 

■ 

la 

*  * 

*  #  * 

B 

* 

Next/Back 

Your  responses  should  reflect  how  strongly  you  be!  ieve  each  group  is  I  ikey  to  attack, 
rounded  to  the  nearest  percent  Your  response  probabi  Sties  must  sum  to  1GQ%.  You 
can  do  this  in  your  head,  or  you  can  fee  the  system  sum  for  you.  if  your  answers  do  not 
sum  to  100%,  clicking  "Next"  wi  II  a  atom  aocaify  adjust  the  probabilities  to  100%  (the 
a  dj  u  stment  v^iV  maintain  the  re  larirve  proptxt  ion  across  all  a  nswers),  For  i  nstance: 


Pi  ease  cl  *ck  "Next"  to  try  i  t  now 


**V 

r**  w* 

IK 

Mr\ 

IK 

■ 

□ 

■ 

I  *  A 

*• 

% 

* 

Next/Back 

You  may  click  "Back"  if  you  wish  to  make  any  adjustments.  When  sat  itf  ied, 
click  "Next”  to  report  These  levels  as  your  final  judgments.  Pi  ease  note  that 
once  you  have  clicked  "Next”  a  second  time,  you  may  not  go  back  to  change 
your  answers. 


ip. 

□ 

■ 

■ 

ID 

*  * 

l«  * 

B 

* 
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Scoring  for  Judging  the  Probabilities  of  Attack 

Each  mission  wit  consist  of  10  *20  instances  or  trials  in  which  you  provide 
judgments  of  attacks. 

At  the  end  of  each  mission,  you  will  be  scoredonhow  well  you  judged 
probate  ties.  Your  judgm  ents  v^il  be  compared  to  the  ca  taxations  of  a  n 
optimal  a  naiyst  that  has  the  same  information  as  you .  Th  isscor  e  w  i  it  be 
averaged  over  m  uit  iple  attacks,  a  nd  w  ill  onty  b  eg  iven  ax  the  end  of  a  M  i  sso  n. 
For  example: 

■  Assume  your  probability  judgments  for  an  attack  a  re: 

{A  =  40%,  0  =  30%,  C  =  20%.  0  =  10%} 

and  the  optimal  ana  FysTs  judgments  are: 

{A  =  10%,  B  =  20%,  C  -  30%,  0  *  40%}, 

Using  a  similar  rty*  based  scoring  system,  your  score  for  just  this  attack  would  be 
63  of  100  points. 

■  if  your  probability  estimates  exactly  matched  the  optimal  analyst,  then  your 
score  would  be  100  of  100  po  t  rrts 


Allocating  Troops 

Afte  r  reporting  probab  i  Stiesof  each  attack,  you  w  ill  be  asked  to  allocate  troops 
aga  instagroupor  to  a  location.  You  will  receive  a  score  based  onthe 
correctnessof  your  troop  allocation  chokes. 

In  M  issions  1*3,  you  will  be  asked  to  choose  a  single  group  or  i  ocatiomo 
a  I  Mrate  all  your  troops  to. 

Try  a  1 1  ocating  troo  ps  against  a  sing  legroup  now  using  the  i  nterface  below 
(select  a  radio  button). 


Allocating  Troops 

In  Mi55ions4-7,YQU  will  be  asked  to  a  I  bcatetroops  in  percentages.  You  can 
allocate  100%  to  one  group,  or  you  can  allocate  some  percentage  to  each  group  as 
long  asthepercent^essumto  100%.  You  wil  enter  your  chokes  using  an 
interface  like  the  one  you  u  sed  to  enter  probabilrtiesof  attacks. 

Try  a  i  locating  troops  against  several  groups  using  the  interface  below. 


Scoring  for  Troop  Allocation 

After  enter  ing  your  responses,  you  v/iltb  e  shown  w  h  ich  gro  up  attacked,  and  will 
receive  a  score.  The  score  is  based  on  the  percentage  or  proportion  of  troops 
you  a  I  locate.  For  example: 

■  Assume  you  a  [locate  troops  against  a  ssngiegroupCMisscms  1  -  3) 

{A  =  100%,  B  »  0%J 

if  th  e  attack  was  by  g  roup  A,  you  would  score  100  out  of  100  po  ints  i  n 
allocating  troops. 

if  the  attack  was  instead  by  group  B,  you  would  scoreO  points, 

■  As  sume  you  allocate  troops  in  percentages  issbn  4  -  7) 

{A  =  40%,  8  =  30%,  C  *  20%.  O  =  10%}. 

If  th  e  anack  w  as  by  group  A,  you  would  score  40  out  of  100  points  in 
a  I  locating  troops. 

If  th  e  attack  was  instead  by  group  8,  you  would  sco  re  30  po  ints. 


Surprise 

At  certain  points  in  the  mission,  you  will  be  asked  to  report  your  degree  of 
surprise  a  bout  a  n  observat  ion,  by  s  electing  fro  rn  a  scale  of  7  opt  sons 
ranging  from  "Not  At  All  Surprised*' to  "Extremely  Surprised". 

N  ate  that,  uni  ike  probability  j  u  dg  ments  and  troop  a  i  locations,  there  are 
no  rigfot  or  wrong  answers  when  it  comes  to  surprise.  PS  ease  report 
whatever  deg  ree  of  surpass  you  f  ee^  at  th  e  moment  you  are  prom  pted. 


H«lJUjk|l  tofrMMp 


Screen  Display  rHa  iw  sve-vew  of  fre  nte^Kt  you  </  se  usjigte  "«e  nformatoo 

a#MI  *£*ee«  $  d  v^3«  nt*  KWVi  a**ai 


Screen  Displ  ay  eac  h  Step  of  each  mission,  you  will  be  give  n  cu  rre  nt  i  nstr  uctions  of  w  hat  to  do  next 

at  the  top  of  the  screen. 


Screen  Display  ^  tan  if?  v^at  th«  ci_— ert  sTuaton  s  *1  tt*  Mutton  A-ea 


View  the 
Miwiort  Are# 
n*-* 
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Screen  Display  A.1  of  i'C’W'  #tt*c*  probate*  *tf  troop  iBooMfis.  and  wph»  -espo-its 

•V  fie  given  on  the  -  gnt  hanfl  *tee  of  the  it  w. 


Ert?f  font 
responses 


Screen  Dis  pi  ay  You  car> click  *  button  to  review  the  mission  tutorial  at  any  time. 

You  can  see  the  current  mission  status  at  the  bottom  of  the  screen. 


View  the 

mission 
status  here 


Review 

the 

mission 

tutorial 

here. 
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Screen  Dis  pi  ay  This  3  complete  example  of  the  display  you  will  see  for  a  typical  mission. 

There's  no  need  to  study  it  now,  since  you  will  become  more  familiar  with  it  later. 


Missions 

During  theexperiment  you  will  beaskedto  complete?  different  missons. 
More  specific  instructions  wilf  be  given  at  the  start  of  each  mission 

You  will  beg  Even  a  break  after  Mission  3,  then  another  break  after  Mission 
5.  You  are  free  to  take  more  breaks,  but  please  try  to  wait  until  the  end  of 
a  mission. 

Please  ask  the  study  proctor  ifyou  need  help  at  arry  ti  me  during  the 
experiment. 

IMPORTANT:  Pi  ease  do  not  use  a  pen,  pencil  paper,  calculator,  or  any 
other  extern  aide  vice.  The  study  is  intended  to  measure  how  wefl  you  can 
perform  without  any  aids. 

Click  Next  to  begin  Mission  1, 


2  Mission  1 


Learning  two  groups.  [10  blocks  of  10  trials].  Attacks  by  two  groups  (A,  B)  are  generated 
(SIGACTS),  one  on  each  trial,  by  two  normal  distributions -each  with  a  different  base  rate, 
center  location,  and  dispersion.  On  the  last  trial  of  each  block  the  subject  is  shown  an  attack 
location  and  asked  to  predict  the  probability  of  attack  by  each  group,  {A%,  B%}.  The  subject 
then  makes  a  forced  choice  (A  or  B),  to  allocate  resources  against  the  attack.  At  the  end  of  the 
trial  the  subject  sees  ground  truth  (A  or  B)  at  the  attack  location  and  he  reports  surprise  (on  a  7- 
point  Likert  scale). 
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Mission  1:  Attacks  by  Two  Groups 

You  will  see  attacks  at  various  locations  in  the  mission  area  One  attack  wiiibe 
shown  each  t  i  me  you  click  *  NexT.  Each  attack  is  caused  by  group  AA  or  0#,  as 
indkated  on  the  screen. 

On  every  10th  atta  ck  you  wii  be  provided  an  attack  location,  but  not  the 
identity  of  the  attackingg  roup  (A  o  r  B) .  You  r  job  *sto  judge  th  e  probability  that 
the  anack  is  by  group  A  or  8. 

You  will  do  this  10  times  as  100  attacks  are  accumulated  on  the  screen. 


Mission  1:  Group  Tendencies 

At  th  e  start  of  the  M  issbn,  you  w  ill  not  know  anything  a  bout  a  group's  attack 
tendency.  By  study ing th e  pattern  of  arracks,  you  w il  grad ualy  becom e able 
to  judg  e  th  e  pro  babifity  of  attack  by  each  group.  Each  group  will  behave 
differently.  For  example: 


■  Each  g roup  hasa  different  operational  center 
that  rem  ains  co  nstant  throughout  M  ission  1. 
The  probability  of  attack  is  highest  near  the 
center,  a  nd  decreases  farther  from  the  ce  nter 
along  a  belt  curve  (see  illustration).  Note:  the 
group  center  y^illnot  bevisibie;  it  must  be 
i  nf  erred  fro  m  th  e  pattern  of  attacks. 

•  One  group  may  attack  farther  from  its  group 
center  than  another  [spread  out  more), 

•  One  group  may  attack  more  frequently  than 
another 


O-frwp 

°*n^  High  Probability 


■You  should  always  judge  based  on  all  the 
attacks  you  have  seen,  not  just  the  most  recent 
attacks. 
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3  Mission  2 


Learning  four  groups.  [5  blocks  of  20  trials].  Unlike  Task  1,  there  are  four  different  groups  (A,  B, 
C,  and  D).  Also,  before  reporting  {A%,  B%,  C%,  D%},  the  subject  is  required  to  draw  four  circles  - 
each  representing  the  "two-to-one  boundary"  for  a  group  (i.e.,  attack  by  the  group  is  twice  as 
likely  inside  the  circle  compared  to  outside  the  circle).  After  reporting  the  probabilities  {A%,  B%, 
C%,  D%}  and  making  a  forced  choice  in  allocating  resources  against  one  group,  the  subject  is 
shown  ground  truth  (A,  B,  C,  or  D)  at  the  attack  location  and  he  reports  surprise. 
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Mission  2:  Attacks  by  Four  Groups 

Mission  2  is  much  fikeMisson  1,  but  now  you  areworking  in  a  completely  different 
area  with  four  new  attacking  groups!  St  iilcailed  B*,  C  D  •). 

As  i  n  Missbnl: 

*  Each  group  has  a  d  Efferent  operations  i  center  that  re  mains  constantthroLghout 
the  m  ission.  The  probabiity  of  attack  is  highest  near  thecemer,  and  decreases 
farther  f  rom  xh  ecenter  a  long  a  bell  curve.  Note:  the  g  roup  ce  nter  will  not  be  visile: 
it  must  be i nferred  from  the  pattern  of  attacks, 

■  One  groupmay  attack  farther  from  rtsg  roup  center  than  another  [spread  out 
more). 

■  One  group  may  attack  more  frequently  than  another. 

•You  should  always  judge  based  on  alt  the  attacks  you  have  seen,  not  just  the  most 
recent  attacks 

You  will  be  asked  to  report  pro  babil  ties  on  every  20^  attack.  You  will  do  this  5  times 
as  IOC  attacks  accumulate  on  the  screen. 


Mission  2:  Continued 

In  Mission  2t  in  addit  io  n  to  reporting  probabi  it  ies,  you  w  Ml  be  asked  to  judge 
the  position  of  each  group's  center  and  its  spread. 

You  will  indicate  your  judgment  of  each  groups'  spread  by  drawing  a  circle  that 
captures  the  2*to*l  boundary  of  attack.  This  circle  encloses  an  area  containing 
2/3  of  the  attacks;  1/3  of  the  attacks  will  be  outside  thecirde. 


High  Probability 
of  Attack 


Low  Probability 
^  of  Attack 


2-to-l  boundary, 
capturing  2/3  of  the 
attacks 
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ICArUS  Test  &  Evaluation  Suite  -  F.naiEjtamjcrW 


File  Feature  Vector  Options  Exam  Options  Help 


•  Review  Mission  Tutorial 


Using  the  radio  buttons,  allocate  troops  to  one  of  the  groups,  and  dick  Next  to  continue 


Layers 

/  Attacks  Q 


Legend 

□  Attacks 

A  Aqua  Attack 
•]  gromine  Attack 
[4]  Citrine  Attack 
♦  Qiamond  Attack 
pTj  Unknown  Attack 


Exam;  Find  Ekam  Participant- TEST  Phase  Mission  2  {2n\  Trial  20/100,  Part  4/6 
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4  Mission  3 


Finding  centers.  [5  blocks  of  20  trials].  Unlike  Task  2,  the  task  is  to  find  group  centers  along 
roads  "as  the  cow  walks"  (rather  than  "as  the  crow  flies").  After  each  block  of  trials,  the  subject 
clicks  four  points  A,  B,  C,  and  D  on  the  roads  -  each  representing  the  "center  of  gravity"  for  a 
group  (i.e.,  maximum  likelihood  location  for  a  group's  center,  given  the  group's  attacks).  The 
subject  then  reports  {A%,  B%,  C%,  D%}  at  the  attack  location.  After  reporting  the  probabilities 
and  making  a  forced  choice  in  allocating  resources,  the  subject  is  shown  ground  truth  (A,  B,  C, 
or  D)  at  the  attack  location  and  he  reports  surprise. 
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Mission  3:  Attacks  on  Roadways 

In  Missions,  you  a  re  once  again  working  in  a  completely  different  area  with  four 
new  attacking  groups  [stil  called  Aa,  8*  c  %  D+}_ 

Mission  3  is  much  likeMisson  2,  except  that  everything  now  happens  on  a  road: 

*  Groups  are  centered  along  roads. 

*  AH  attacks  will  appear  on  or  near  roads. 

You  should  always  judge  based  on  a  II  the  attacks  you  haveseenH  not  just  the  most 
recent  attacks. 

On  every  20^  attack,  you  will  see  the  i  ocatkxi  of  attack  by  an  unknow  n  g  roup  and  be 
asked  to  report  the  pro  babil  fey  of  attack  by  each  group.  You  will  do  this  5  ti  m  es  as 
100  attacks  accumulate  on  the  screen. 
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5  Handbook 


Handbook:  PROBS  and  UMTS 

Missions  4-7  a  re  very  different  from  Missions  1-3.  Rather  than  acquiring  data  on 
indMduaiatiscks over  time,  your  knowledge  of  groups'  attack  patter  ns  comes  from 
an  intelligence  Handbook  cal  led  PROBS  (short  for  probabilities).  PROBS  providesyou 
wi  thruiesfor  howto  interpret  data  from  various  intelligence  sources  (INTS). 

I  NTS  come  in  five  types  and  provide  as  to  which  group  may  have  attacked  or 

where  th  ey  will  I  ikety  arack  in  a  g  iven  trial 

1*  HUMINT:  Human  intei  If  ence  re  ports  the  location  of  each  group  center. 

I .  IMINT:  I  mage  intelligence  reports  the  type  of  building  at  a  location. 

3.  MOVINT :  Movement  i  ntel  ig ence  re  ports  the  density  of  traffic  at  a  location. 

4,  SIGINT.  Signaii  mergence  reports  a  group's  use  of  cellphones, 

5-  SOCINT:  Socio-cult  ural  intelligence  reports  the  groups'  favored  regions. 
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Using  PROBS 

You  should  employ  the  PROBS  Handbook  throughout  Mls5ions4-7  because 
you  will  need  it  to  understand  and  make  judgments  from  the  I  NT  data. 

The  PROBS  will  appear  in  a  separate  pop-up  window  whenever  you  receive 
I  NT  data,  feel  free  to  move  the  PROBS  window  to  wherever  you  can  see  rt 
best,  You  can  access  the  PROBS  Handbook  at  anytime  by  clicking  PROBS  on 
the  left  of  the  screen. 


Click  'Back  / 
Ne*l'  to  cycle 
through  the 
(NTS 


Using  PROBS 

Cl  ick  thePRQBSflmknow  to  openthe  PROBS  Handbook  while  you  review 
the  following  pages.  Then  cl  kk  'Next  to  continue. 
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Generic  Constraints 

There  a  re  two  I  NTs  that  apply  to  all  groups  These  are  HUM  1  NT  andSOCINT. 

•  for  HUMINT,  the  probabitfy  of  attack  decreases  as  the  attack  location  gee 
farther  from  a  groups  center  (as  measured  a  long  roads).  St  decreases  with 
distance  along  a  ben  curve  like  in  Mission  3. 

■  For  SQGNT,  the  probability  of  attack  by  a  g roup  i s  Sarger  ( by  a  f actor  of  2* 
to-1)  in  the  group's  own  region  compared  to  each  other  group's  region. 


Specific  Constraints 

The  PROB5  for  I  MINT  and  MOVINT  capturetendenciesthat  are  specific  to 
each  group.  For  example: 

■  For  f MINT,  groups  A  and  B  prefer  (by  a  factor  of  4-to-l)  to  attack  near 
government  buildings  as  opposed  to  military  buildings,  and  vice  versa  for 
groups C  andO. 

■  For  MOVINT,  groupsAandC  prefer  to  attack  (by  a  factor  of  4-to-l)  in 
dense  traffic  as  opposed  to  sparse  traffic,  and  vice  versa  for  groupsB  andO. 


The  PRQB5  for  5IQNT are  d  ifferem  becauseSlGlNT  will  report  on  only  one 
group, 

•  WhenSIGINT  on  a  group  reports  cel!  phone  activity  (chatter),  attack  by 
that  group  is  more  likely  than  each  other  group  (by  a  factor  of  7- to-1).  When 
SiGINTonagroupreportssilence,  attackbythatgroupisleas  ;ikefy(bya 
factor  of  l-to-3)  than  each  other  gro  up. 


INTS  are  Independent 

I M  PORTA  NT:  The  I  NTS  you  receive  provid  e  all  the  information  you  need  to 
perfectly  judge  the  probability  of  attack  in  Missions  7. 

IMPORTANT:  Each  I  NT  source  provides  an  independent  clue  about  which 
g  rou  p  co  nducted  an  attack,  SpecrfKaiy,  th  e  I  NT  data  you  see  at  a  ny  ti  me 
depends  o  nfy  o  n  theg  roup  that  attacks  a  nd  is  not  affected  by  w  h  at  the 
Other  l  NTS  say 

For  example,  the  type  of  traffic  (MOV!  NT)  »snot  affected  by  the  type  of 
nearby  building  (IMtNT),  so  the  fikelhood  of  seeing  an  attack  in  densetraffic 
near  a  government  building  depends  only  on  the  group  that  attacks,  it  does 
not  depend  on  the  I  ikefihood  of  seeing  densetraffic  given  a  government 
building, 

D  ick  Next  to  begin  Missions 


6  Mission  4 


Scoping  attacks.  [10  trials,  2  stages  per  trial].  Unlike  Task  3,  the  subject  is  given  the  center 
(from  HUMINT)  for  one  group,  along  with  four  possible  locations  1,  2,  3,  and  4  of  attack  by  that 
group.  In  stage  1,  the  subject  is  required  to  estimate  the  probability  of  attack  at  each  location 
based  on  a  normal  distance  decay  function  along  roads.  In  stage  2,  the  subject  is  given  the 
regional  boundaries  for  groups  (SOCINT),  along  with  the  inside/outside  region  attack  likelihoods 
from  PROBS.  The  subject  is  required  to  aggregate  the  prior  (HUMINT)  and  SOCINT  probabilities. 
The  subject  allocates  resources  in  percentages  (not  a  forced  choice)  across  the  sites,  and  is 
shown  ground  truth  (1,  2,  3,  or  4).  The  subject  reports  surprise  after  observing  ground  truth. 
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Mission  4:  One  Group  and  Four  Attack  Locations 


In  Mission  4,  the  attacking  group  is  known  to  beD,  but  the 
attack  may  come  at  one  of  four  locations (1,2,  3,  or 4),  Your 
fob  is  to  judge  the  probability  of  attack  at  each  location. 
You  will  haveaccestotwo  intelligence  sources  (1NTS)  in 
this  miss  on. 

Your  first  step  is  to  estimate  the  probability  of  attack  at 
each  location  based  on  data  from  HU  MINT  and  the 
know  ledge  contained  inyour  l  nteiljgence  Handbook 
(PR08S)  HUMINTillustratesthefocationof  D's  group 
center,  a nd PRO BS  for  HUMINT  specifies  how  the 
probability  of  attack  depends  on  the  distance  (along 
roads}  from  the  group  center. 


Example  map 


Mission  4:  One  Group  and  Four  Attack  Locations 

After  reporting  probab i fit  ie^  you  w ill  beg iven  another c I ue 
in  the  form  of  SOCi  NT  d  ata.  The  probability  of  attack  in 
the  group's  own  region  is  twice  as  high  compared  to 
attacks  in  another  group's  region. 

You  wi  II  be  asked  to  report  your  surprise  upon  receivirtg 
the  SOCINT  data.  For  example,  after  HUMINT  you  may  think 
location  "3"  is  the  most  HkeV  location  of  attack,  in  that  case 
you  may  be  surprised  if  SOCINT  reportsthat  location  wy  is 
outside  group  DJs  preferred  region  of  attack. 

Next,  usingSOCINX  you'll  be  asked  to  update  your  previous 
judgment  [the  probability  of  attack  at  each  location),  which 
was  based  only  on  HUMINT  Your  up  dated  judgment 
should  consider  both  HUMINT  and  SOCINT 


Example  map 


HUMINT  and  SOCINT 
shown 


Mission  4:  Continued 

After  reporting  your  updated  pro  babil  ties,  you  wii  b  e  asked  to  ai  locate 
troops  m  percentages,  Youcan  allocate  any  percent  age  of  troops  to  a 
location,  but  the  total  troop  allocation  over  the  four  Jocationsmustsumto 
100%. 

You  v/ 1 II  be  asked  to  do  this  10  times  Each  trial  is  independent;  that  is,  each 
attack  wii  occur  in  a  new  area  of  interest,  with  d  Efferent  roads  and  attack 
locations. 
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[C4rfJS  Ten  &  EvaiiMtjen  Suite  -  F irwl  EjwnusrW 


Fir*  t  omiiNj  Vector  Opiitwi-,  Exam  Oprions  Help 


Eteck 


■  Review  Mission  Tutorial 


The  SOCINT  layer  if>  now  shown  Indicate  Ibe  n?dated  probability  ihat  the  attack  will  occu  at  each  location,  and  click  Next  to  continue 


J  Roads  ^ 

J  Locations  ^ 
J  HUMIMT 
J  SOCIHT 


your  updated  protutMliU+s 

2«>  ZKi  25\ 


Exam.  Final  Exam  Participant,  TEST  Phase  Mission  4  (4/7}  Trial  1/10.  Part.  3/S 
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7  Mission  5 


Fusing  layers.  [10  trials,  4  stages  per  trial].  Unlike  Task  4,  the  subject  is  given  an  attack  location 
(SIGACT)  along  with  group  centers  (HUMINT)  for  four  groups  (A,  B,  C,  D).  The  subject  is  also 
given  the  probabilities  of  attack  by  each  group  {A%,  B%,  C%,  D%},  based  on  the  HUMINT 
distance  function.  The  subject  is  then  given  four  more  INTS  (IMINT,  MOVINT,  SIGINT,  SOCINT), 
one  at  a  time,  along  with  the  likelihoods  (from  PROBS)  for  features  of  INTS.  At  each  stage  the 
subject  updates  probabilities  {A%,  B%,  C%,  D%}.  After  the  last  stage  the  subject  allocates 
resources  in  percentages  {A%,  B%,  C%,  D%}  against  the  groups.  At  the  end  of  the  trial  the 
subject  is  shown  ground  truth  and  he  reports  surprise. 
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Mission  5:  Four  Groups  and  One  Attack  Location 

in  M  i  ssion  5,  th  er  e  will  be  a  single  attack  location  and  four  possible 
attacking  groups  (still  called  C*  D+)~  Nov/  you  will  be  given 

more  INT  data  about  each  attack. 

HUMINT  wilt  report  the  locations  of  groupcenters,  as  in  Mi  ssion  4.  But  in 
Mission  5  probabilities  will  be  auto  matkatty  computed  usng  the  HUM  I  NT 
d  i  stance  fund  joa  These  calc  ui  at  sons  are  exact. 

Subsequently  you  will  see  data  from  four  more  I  NTS  -  on  eat  a  time,  but 
in  an  order  that  will  change: 

IMINT 

MOVINT 

5IGINT 

SOC1NT 


Mission  5:  Continued 

After  receding  a  new 1  NT  you  wil  be  asked  to  revise  your  previous 
probabiltYjudgmentsinlightofthenew  INTr  taking  into  consideration  the 
new  INI  and  oft  previous  INTs. 

You  w  i  IJ  be  asked  to  do  th  is  10  ti  mes  Each  trial  is  i  ndep  erdent;  that  ts# 
each  attack  will  occur  in  a  new  area  of  interest;  with  different  roads  and 
attack  locations. 
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8  Mission  6 


Choosing  layers.  [10  trials,  3  stages  per  trial].  Much  like  Task  5,  except  the  subject  decides 
which  layer  of  INT  (IMINT,  MOVINT,  SIGINT,  or  SOCINT)  to  receive  on  each  of  three  stages,  after 
receiving  the  initial  HUMINT.  The  criterion  for  each  choice  is  to  maximize  the  expected  amount 
of  useful  information  to  be  gained  from  the  choice  of  an  INT. 
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Mission  6:  You  Choose  I  NTS 

Mission  6  is  much  like  Mission  5,  except  that  after  HUM  INT  is  displayed, 
you  wit!,  choose  any  3  of  4  INTS  from  a  menu,  one  at  a  time. 

After  you  select  an  INI  it  is  no  longer  available  for  selection  from  the 
menu.  Rem  ember  th  at  SlCIISTT  is  obtained  on  a  single  group  of  your 
choice.  After  choosing  to  obtain  SIGINTon  one  group,  SIGN!  will  no 
i  o  ng  er  b  e  a  va*  able  on  that  group  o  r  a  ny  other  gr  o  up. 

Vou  shouidchooselNTSat  each  step  to  maximize  the  amount  of 
information  you  expect  to  receive  for  judging  attack  probabt  it  ies  across 
groups. 

You  wi  tl  be  asked  to  do  this  10  times  Each  trial  is  i  ndepeodent;  th  at  is, 
each  attack  v/ ill  occur  in  a  new  area  of  interest,  with  different  roads  and 
attack  locations. 
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9  Mission  7 


Self-initiated  sensemaking.  [15  trials].  Task  7  is  a  combination,  in  part,  of  Tasks  4  and  6,  but 
with  a  focus  on  self-initiated  sensemaking.  Participants  will  be  given  the  centers  of  four  groups 
(A-D)  and  four  possible  attack  sites  (1-4).  As  in  Task  4,  the  primary  task  is  to  predict  the  location 
of  attack  and  to  allocate  resources  to  each  location  {@1%,  @2%,  @3%,  @4%}.  But  unlike  Task 
4,  participants  will  decide  if  and  when  to  seek  additional  data,  as  well  as  which  data  (INT)  to 
seek.  See  the  Phase  1  Challenge  Problem  Design  and  Test  Specification  document  for  a  more 
detailed  description  of  Task  7,  including  a  summary  of  the  major  structural  differences  between 
Task  7  and  Tasks  1-6. 
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Mission  7:  Groups  Attack  in  Waves  Over  Time 


M  isskxi  7  d  iffersf  rom  m  issbns  4-6  i  n  f  our  w  ays: 

1.  in  Mission?, you  will  be  working  in  as^ngleareafora 
long  period  of  time-  the  map  won't  change  from  tria  l 
to  trial. 

2.  The  map  js  more  complex.  Each  map  shows  4  groups 
and  4  ca  ndidate  attack  I  ocat  ions.  Your  task  is  to  report 
which  group  is  I  ikely  to  attack  at  which  location,  and  to 
a  I  locate  troops  to  counter  the  attack. 


3.  You  may  use  credits  to  purchase  some  INI  types. 

4.  You  have  access  to  new  I  NT  k  now  i  ecfee:  groups  attack 
in  waves  (click 'Next  to  learn  about  this  new  rufej. 


Mission  7:  Wave  Rule 

We  have  new  inteiiigencethat  groups  tend  to  attack  in  waves.  If  a  group  attacked 
in  the  previous  trial,  then  they  are  S5%  likely  to  attack  again  in  the  current  trial. 
This  'wave'  rule  is  now  mduded  inyourrufebook. 


If  A  attacked  in 
the  last  trial 


A  B  €  D 

A  •  I  ♦ 


If  B  attacked  in 
the  last  trial 


A  B  C  0 

A  •  I  ♦ 


If  C  attacked  in 
the  last  trial 


A  B  C  O 

A  *  *  ♦ 


If  D attacked  in 
the  last  trial 


A  B  C  D 

A  •  A  ♦ 


Mission  7:  Trial  Walkthrough 


Step  1.  At  the  beginning  of  each 
trial  you  will  be  asked  to  report 
which  group  is  likely  to  attack 
next  using  the  dues  that  the 
INIS  provide. 

You  are  not  required  to  report 
the  probability  of  attack  by  each 
group  on  each  trial,  If  you 
choose  to  make  a  report,  click 
"Update  Group  Judgment/'  If 
you  choose  not  to  make  a 
report,  click  "Next/' 


Mission  7:  Trial  Walkthrough 


Step  2.  Next,  you  will  be  asked  to 

report  which  location  is  likely  to 
be  attacked. 

Note:  HUMINT  and  SOCINT  are 
provided  "for  free" 


Mission  7:  Trial  Walkthrough 


Step  3,  You  will  be  asked  to 

allocate  troops  to  each  location. 


Mission  7:  Trial  Walkthrough 


Step  4,  Next,  the  actual  attack 
location  will  be  revealed . 

If  the  attacker  is  known,  it  will 
appear  Inside  the  attack  location; 
otherwise,  a  ted  '?'  will  appear, 
indicating  the  attacker  was 
unknown. 

Note:  The  last  5  attack  locations 
will  remain  on  the  screen  at  all 
times. 
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Mission  7:  Trial  Walkthrough 


Step  5.  You  will  receive  credits 

based  on  how  effectively  you 
allocated  troops. 

You  can  use  credits  to  purchase 
additional  intelligence  when 
prompted.  Intelligence  can  be 
used  to  get  back  on  track  if  you 
incorrectly  predicted  the  attacker 
or  attack  location. 


Mission  7:  Trial  Walkthrough 


After  viewing  the  Intelligence 
you  purchased,  you'll  be  asked 
to  update  the  probability  of 
attack  by  each  group,  and  a  new 
trial  will  start. 

Mission  7  will  last  for  15  trials. 


Mission  7:  Notes  about  Credits 

You  have  a  limited  number  of  credits  to  spend,  so  use  them  w  isely  You  v/ill 
start  The  mission  with  10  credits. 

You  may  purchase  as  many  layers  as  you  have  credits  for.  Each  layer  costs  1-2 
credits, 

Al  l  credits  expire  at  th  e  end  of  the  m  ission.  You  wii  not  receive  a  h  igh  er 
score  if  you  have  credits  remaining. 


Mission?:  Beginning  and  Ending 

On  tri  all,  you  will  start  m  the  middle  of  an  attack  wave.  At  step  1  of  the 
trial  you  will  be  told  which  group  attacked  just  before  you  were  assgned 
to  the  missoa  a  nd  the  pr obabil  ties  will  a  utomatjcaiiY  pre-  populate 
accordingtothenew  waveruie. 

Yo  u  can  cl  kk  'Review  M  ission  Tutorial  at  the  top  of  theGUI  w  indow  to 
review  this  tutorial  at  any  time  during  them  isson. 


Q0  ICA/US  Ten  &  Evaluation  Suite  -  Finattxairvwnl 

File  Feature  V/oclof  Option-  Exam  Options  Help 

<  [lock  Next  ->  -  Review  Iliaaion  Tutorial 


The  attack  occured  ai  Location  1 ,  and  the  group  responsible  is  not  known  (?)  Your  received  0.25  credit*  on  this  trial  tor  allocating  troops  Click 

Next  to  continue 


Layers 

J  Roads  ^ 

J  Locations  Q 

J  HUMINT  2  (PROBSl 
)  SOCINT  J  /PROBS1 


0 
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your  troop  allocations 
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your  previous  probabilities 
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10  Appendix  A:  PROBS 


HUMINT:  Human  Intelligence 


if  d  group  attacks,  then  the  relative  likelihood  of  attack 
decreases  as  the  distance  from  the  group  center  increases, 

•The  likelihood  of  attack  is  about  40%  at  the  group  center 
■  The  likelihood  of  attack  i s about  35 %  at  S  miles. 

•The  likelihood  of  attack  is  about  2554?  at  10  miles. 

■The  likelihood  of  attack  is  about  5%  at  20  miles. 

•The  likelihood  of  attack  is  nearly  Ofrfeat  40  miles 


IMINT:  Image  Intelligence 


Ifthe  attack  is  near  a 
government  building, 
t  hen  attack  by  A  or  B 
is  4  times  as  likefv  as 
attack  byCorO. 

■ 


A  B  C  D 
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If  the  attack  is  near  a 
military  building, 
then  attack  by  Cor  D 
is  4  times  as  likety  as 
attack  byAorB. 


A 


A  0  C  0 

A  *  4  ♦ 
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MOVINT:  Movement  Intelligence 

If  the  attack  is  in 

If  the  attack  tsin 

dense  traffic,  then 

sparse  traffic,  then 

attack  by  A  or  c  is 

attack  by  B  or  0  is 

4  times  as  likely  as 

4  times  as  likely  as 

attack  by  B  or  D. 

attack  by  A  or  C 

a] 

m 

!■!□ 

A  B  C  D 

A  6  C  0 

4  8  4  4 

4  8  4  8 

StGINT:  SIGNAL  Intelligence 


If  SIGIISTT  on  a  group  re  ports  chaffer,  then  attack  by  that  group  is 
7  times  as  likely  as  attack  by  each  other  group. 


^Chatter  Chatter  <0)  Chatter  Chatter 


If  SIGIIMT  on  a  group  re  ports  silence,  then  attack  by  that  group  rs 
1/S  as  likely  as  attack  by  each  other  group. 


®  Silent 


A  B  C  0 

4  8  4  4 


W  Silent 
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©  Silent 
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'  T  Silent 

Silo 
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SOCINT:  Socio-Cultural  Intelligence 

If  the  a  nack  is  in  a  groups  region,  then  attack  bv  that  group  is 
2  times  as  likely  as  attack  by  each  other  group. 
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Region  8 

■ 
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— 

Region  0 
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A  B  C  D 
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A  *  «  ♦ 
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"Wave”  Rule 

Ha  group  attacked  in  the  o  revioustr  i  al  then  they  are  85% 
I  ijjceiv to  attack  again  in  th ecu rrent trial 


H  A  attacked  in 
the  fast  trial 


A  B  C  D 

a  e  4  ♦ 


tf  e  attacked  in 
the  last  trial 


I 


A  B  C  0 

A  •  I  t 


If  C  attacked  in  If  D  attacked  in 
the  last  trial  the  last  trial 


A  BCD 

A  •  %  + 


a  e  c  o 
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